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ABSTRACT 



This evaluation manual is a response to requests of 
education and training organizations for a practical methodology with tools 
to evaluate information and communications technology (ICT) projects and use 
of information and learning technologies in teaching and learning in Great 
Britain. Section 1 describes evaluation methodologies, the evaluation tools 
in the form of a set of three customizable questionnaire templates, and how 
to manage the evaluation process. Section 2 contains three customizable 
questionnaires designed to collect feedback from learners, staff supporting 
learners, and managers. The questionnaire templates are designed primarily 
for use in evaluating individual projects, strategies, or specific ICT 
learning and teaching but can also be used to make comparisons within or 
between organizations. Section 3 discusses customizing the questionnaire 
templates . Section 4 provides information on evaluation methodologies . 
Section 5 contains a customizable analysis guidance template with questions 
to consider when analyzing responses to the questionnaires, developing 
reports, and developing action plans. Appendixes contain information on six 
colleges involved in trials of the templates, 25 references, and glossary. 
(YLB) 
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Summary 



This evaluation manual has been produced in response to requests from education 
and training organisations for a practical methodology with tools that can be used to 
evaluate information and communications technology (ICT) projects and/or the use 
of information and learning technologies (ILT) in teaching and learning. 

It describes relevant evaluation methodologies and provides evaluation tools in 
the form of a set of three customisable questionnaire templates. These templates can 
be used to develop questionnaires to collect the observations, experiences and opinions 
of managers, staff (especially teaching staff) and learners, as these three groups are likely 
to be the major stakeholders in the introduction of ICT into teaching and learning or 
the implementation of ICT projects. 

The questionnaire templates have been developed, drawing on research evidence 
and practical experience at the Open University (OU) and trials of draft templates 
in multiple projects within further education (FE) colleges. 

It is essential to customise the questionnaire templates to incorporate specific and 
relevant questions about the unique aims and objectives of the activities being evaluated 
and to ensure that the needs of specific groups of stakeholders are addressed. 

The evaluation tools provided have been primarily designed to evaluate individual 
projects, strategies or specific uses of ICT in teaching and learning. They can also be 
used to compare different projects, strategies or uses of ICT within or between 
organisations, provided care is taken to ensure like-with-like comparisons. 

The manual also contains a customisable analysis guidance template with questions 
to consider when analysing responses to the questionnaires, reports and action plans. 



Introduction 



1 



Background and intended use 

This manual has been produced in response to requests from education and training 
organisations for a practical methodology with evaluation tools. It can be used to evaluate 
specific information and communications technology (ICT) projects and the use of infor- 
mation and learning technologies (ILT) in teaching and learning. It will also be useful 
for developing ICT materials and implementing self-assessment processes. 

The evaluation tools are questionnaire templates that have been designed primarily 
to evaluate individual projects, strategies or specific uses of information and communi- 
cations technology in teaching and learning. They can also be used to compare different 
projects, strategies or uses of information and communications technology within 
or between organisations. However: 

• it will be essential to customise the questionnaire templates so that they are 
relevant to the aims and objectives of the specific activities being evaluated 
and to the specific needs of the groups of stakeholders (see Section 3) 

• if the customised questionnaires are used for intra or inter organisation 
comparisons, care must be taken to compare only aspects of projects or uses 
of information and communications technology that are really comparable. 

This manual is not designed to evaluate specific software packages or learning materials. 
There are already several good software evaluation instruments available for this. 

See The right tools for the job: evaluating multimedia, flexible and open learning 
materials (Lockitt, 1999). 

A draft manual was produced by a team of researchers from the Open University 
(OU) working with FEDA. The questionnaire templates originally consisted mostly 
of open-ended questions. FEDA amended these to include more closed questions to 
facilitate quicker and easier completion and analysis of questionnaires. This should 
also make the questionnaires more suitable for cross-college comparisons (see above). 

This manual will be used as part of the National Learning Network (NLN) evaluation 
of the efficiency and effectiveness of investment in information and learning technology 
by the FE sector. Investigations within this evaluation will focus on cohorts of learners 
studying with a representative sample of colleges over two years. Feedback from this 
study will inform further development of the tools provided here. 
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Evaluation methodologies 

There is a substantial body of research evidence available to guide selection and appli- 
cation of evaluation methodologies (see Section 4). The methodology in this manual 
concentrates on surveying project or activity stakeholders through questionnaires. 
Questionnaires are a very familiar method of collecting data, including feedback, 
and have the following advantages: 

• they can be used off the shelf or easily customised 
for a particular context or audience 

• they are relatively quick to implement 

9 they are relatively quick to analyse 

• they can be used for a wide range of projects 
with very different aims and objectives 

• they can be used for formative or summative evaluation 
and for pre~ and post- activity or project evaluation. 

It could be argued that using questionnaires only might result in too much concentration 
on the product rather than the process of learning with ICT. The most thorough types 
of evaluation methodology use data gathered in a variety of ways, including interviews, 
observations (human or video camera), surveys, user and learner diaries and log books, 
pre- and post- activity or project knowledge quizzes and focus groups. 

Some interactions between people and technology are best studied through observation. 
As this can be a resource-hungry process, the staff and learner questionnaires include 
questions designed to highlight these interactions. Comparisons of data collected by 
observation or records kept by learners during learning activities, with data collected 
via questionnaires, can be very illuminating. Users of this manual may therefore wish 
to consider adopting other methods of data collection, as well as the questionnaires. 

If observation is appropriate, it could involve external evaluators or rely on the reflective 
abilities of the staff involved. Some researchers suggest that it is difficult for practitioners 
to reflect objectively on their own practice (Phillips, 1988), but others disagree (Kerry, 1988). 
It is probably true that staff are encouraged in their reflections if the circumstances are 
safe and they feel able to trust the individuals who will see their evaluations. 

Information collected via questionnaires based on these templates may be enriched by, 
and should be compared with, any ‘hard’ data available. If integrated learning systems 
(ILS) or managed learning environments (MLE) are used, these systems provide records 
of what the users have done plus information on their progress and achievements. 

Hard data from student tracking systems, college or management information 
systems (C/MIS), e.g. retention rates and achievement data, can also contribute 
to evidence of the impact of an ICT project or strategy. 
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The questionnaire templates 

A set of three questionnaire templates has been developed for colleges to customise. 

They draw on the research described in Section 4 and practical experience at the 
Open University and draft templates have been trialed in multiple projects in 
six FE colleges (see Appendix 1). 

The questionnaire templates are designed to collect the observations, experiences 
and opinions of learners, staff (especially teaching staff) and managers respectively, 
as these three groups are likely to be the major stakeholders in the introduction of 
information and communications technology into teaching and learning and the 
implementation of ICT projects. 

The involvement of teaching staff in the evaluation process is particularly 
important, as research indicates that many education innovations have failed 
because they underestimated the importance of teaching staff in implementation. 

The format of the staff and manager questionnaire templates is a modified version of 
the OU framework described in Section 4. The templates are divided into three sections: 

• Context: information on the projects themselves and the institution, 
plus information on the commitment and expertise of managers and staff 

• Resources, access and support: information on the computing facilities, 
hardware, software, funding and student support 

• Outcomes: information on the strategic and specific aims of the project. 

The staff and manager questionnaire templates have been designed to follow the 
same overall framework so that they can be easily cross-referenced during analysis. 

The learner questionnaire may be customised to include questions that test relevant 
knowledge held or gained before and/or after a project or use of a particular system 
or software (see Section 3). These additional questions provide a mechanism for 
checking that learning and retention of knowledge have taken place. 

The questionnaire templates are largely composed of ‘closed’ tick box questions plus 
a few open-ended questions. Closed questions are used because they are generally quick 
to complete and easy to analyse. Open-ended questions are used because although they 
are harder to answer and analyse, they give richer feedback on factors that might 
influence the effectiveness of the project or strategy under evaluation. 

In the learner questionnaire, a mix of closed and open questions helps to take 
account of the variable literacy levels of the learners. Also, the potentially varied aims 
of ICT projects and strategies mean that it has been difficult to design specific, closed 
questions about the learners’ experiences for inclusion in a generic questionnaire template. 
For example, during the college-based trials, one learner answered the question ‘Did the 
system help you to learn?’ with ‘No’ but answered the following question, ‘Can you give 
reasons for your answer?’ with ‘It just gave me the information that I was after quickly.’ 

All the questionnaire templates include some statements about a project, or use of 
a particular system/software, and respondents are asked to indicate the extent of their 
agreement or disagreement. Some researchers argue for including both positive and 
negative statements to avoid any bias which may arise if a questionnaire is perceived 
as having a positive or negative tone. However, in the college trials, some learners 
were confused by this approach and so the questionnaire templates provide 
consistently positive statements. 
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Managing the evaluation process 

Evidence from trialing the questionnaire templates and established practice in any 
situation requiring the management of change, indicate that the whole evaluation 
process needs to be overseen by one manager: the evaluation manager. The evaluation 
manager needs to be familiar with the aims and objectives of the ICT project, or strategy, 
and able to decide how the questionnaire templates need to be customised. 

Senior management instigating an evaluation process should consider whether it 
is appropriate for the manager of an ICT project to be responsible for its evaluation. 

This approach would be consistent with self-assessment, which is established practice 
in further education. However, there is also the risk of deliberate enhancement of 
evaluation results and the possibility of a lack of objectivity which is inherent in 
any self-assessment process. 

The evaluation manager’s responsibilities include: 

• deciding how questionnaire tempiates need to be customised 

The two main reasons for customisation are: 

O the type of language used and its suitability for particular learners 
o the need to include questions directly relevant to the 
unique circumstances of the project or strategy. 

In the trials of the questionnaire templates customisation emerged 
as a key factor in determining the success of the evaluation process 
(see Section 3 for further details) 

• ensuring that everyone on the team is aware of the strategic aims 

of the project and knows how to fill in the ‘aims’ section of the staff and 
management questionnaires. The trials revealed that some people found this 
section quite difficult and that team discussion sessions beforehand helped 

• agreeing when the evaiuation shouid be carried out The evaluation may 
occur once, be repeated at key stages of a project or strategy implementation, 
or occur both before and after. Reference to the evaluation tools at the start 
is particularly useful as it can help to: 

O clarify aims and objectives 

O inform development of success criteria and performance indicators, 
if these are appropriate and practical 
o shape subsequent development 

• deciding if sampiing is appropriate and, if so, what sampiing strategy to use 

e.g. how many staff, and how many learners, should be asked to complete 
questionnaires and what criteria should be used to select these samples 
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• coordinating the anaiysis Analysis of the results should take place in four stages: 

o recording and summarising learner responses to their questions 
(e.g. 75% of learners indicated that they ‘enjoyed’ or ‘very much enjoyed’ 
the project). This analysis is often done by computer 

o recording and summarising management and staff responses: if there are 
only a fevv^ questionnaires to analyse, some evaluation managers prefer to 
do it by hand to immerse themselves in the data 

o consideration of the implications of information provided (see Section 5) 

O triangulation^ involving comparison of the responses to similar questions 
by each of the stakeholder groups, provides additional data or highlights 
areas for further investigation or action e.g: 

O did management perceive staff as ‘regular and confident’ users of infor- 
mation and communications technology while staff considered themselves 
‘beginners or occasional users’? If so, it could prompt the question ‘might 
additional staff training have made the project or strategy more successful’? 
The answer will contribute to planning for future projects/strategy 

O did staff believe that most learners ‘really enjoyed’ using the software 
while the learners themselves mostly reported that they ‘do not enjoy’ 
using computers? In this case was the specific system so user friendly, 
or the support provided by staff so good, that it helped people to overcome 
their general dislike of computers? or did the staff not spend enough time 
with the users to be able to gauge their feelings accurately? 

For further help with analysis see Section 5, ‘Analysis guidance template’ For advice 
or assistance with data collection and analysis contact FEDA Survey Research 
Services, tel: 020 7840 5316 or e-mail: gknight@feda.ac.uk 

• deciding whether to collect information in other ways, or from other sources, 

for comparison with or enrichment of information collected via the questionnaires 

• deciding how to feed back the findings of the evaluation process 
to the project stakeholders 

• formulating an action plan to address issues illuminated by the evaluation process 
and to inform future projects or development. 





Introduction 5 



Key messages from the trials 

• Customising the questionnaire templates is vital. Most important is customisation 
to include questions about the unique aims and objectives of the project concerned. 
Some colleges used the questionnaire templates without customising them, which 
led to problems at the analysis stage when it became clear that key questions had 
not been asked. The option to include or leave out questions is also important, 

as sections which some colleges found valuable were of limited importance for 
others. For example, one college suggested that the questions about access to 
computers outside the college were ‘a waste of time’, because all students had 
their own computers. In contrast, at another college very few of the students 
had computer access outside and this question revealed that home access had 
an important influence on the students’ IT literacy and the success of the project. 

• Lack of time to implement projects and strategies was a major constraint 

on customisation. It was clear throughout the trials that colleges were 
finding it difficult to keep to their implementation timetables. 

• The process of thinking through the project or strategic aims is helpful. 

Several managers commented that projects or initiatives are often introduced 
too rapidly to allow time for proper reflection on exactly what they are trying to 
achieve. This observation was supported by the fact that staff sometimes left the 
aims and objectives section of the questionnaire blank. The manager template 
therefore includes questions on whether the staff have been told what the aims 
and objectives are, as well as seeking their opinions on how far these have 
been achieved. 

• Good communication between managers and lecturers and a sense of shared 
ownership of the projects and strategies resulted in the most thoroughly completed 
staff questionnaires. In colleges where communication was less good, the lecturer 
questionnaires tended to be completed rather half-heartedly. Some staff were resis- 
tant to using information and communications technology in their teaching and, 
again, poor communication and a lack of a sense of ownership seem to be at 

the root of this. 

• Inspection and self-assessment preparation were going on at several of the 
colleges during the trials and those colleges used the evaluation tools to 
identify areas of strength and weakness and to collect inspection evidence. 
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The questionnaire 
templates 




This section contains three customisable questionnaires 
designed to collect feedback from: 

• learners 

• staff supporting learners 

• managers. 

They can also be found on the CD accompanying this publication where 

they are supplied in Microsoft Word 98 (as they appear on the following pages) 

and Microsoft Word 95 (for those who do not have access to Microsoft Word 98). 

The questionnaire templates have been designed primarily for use in evaluating 
individual projects, strategies or specific ICT learning and teaching but can also 
be used to make comparisons within or between organisations, as long as care 
is taken to compare like with like. 

It is vital to customise these templates, as different questions have more or less relevance 
for different users. Many of the questions are generally phrased and need to be reworded 
to apply to specific subjects, aims and techniques (see Section 3). 
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Learner questionnaire 
template 



Introduction 

This questionnaire is desiigned to help us to evaluate a project or to look at 
how we use computers in our teaching. It is anonymous and will not be used 
to assess you in any way. 

1. Programme title? (e.g. second year A-level Maths, 

GNVQ Intermediate Science) 



2. Which age group are you in? 



2.1 


14-15 


□ 


2.2 


16-19 


□ 


2.3 


20-25 


□ 


2.4 


26-35 


□ 


2.5 


36-45 


□ 


2.6 


46-60 


□ 


2.7 


Over 60 


□ 



Your use of the system 

This section is designed to help us find out how useful and easy you found 
the system and how well it fitted in with your other studies. 

3. Approximately how many hours have you spent on the system? [ ] 

4. Please tick to indicate whether you agree or disagree with the foiiowing 
statements about the system you used: 







Agree 


Not sure 


Disagree 


4.1 


1 found it easy to get access 


□ 


□ 


□ 


4.2 


1 found it easy to find 
my way around 


□ 


□ 


□ 


4.3 


1 found it easy to get the 
information 1 needed 


□ 


□ 


□ 


4.4 


1 did not have any 
technical difficulties 


□ 


□ 


□ 
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5. Please tick to indicate whether you agree or disagree 
with the following statements: 

Agree Not sure Disagree 



5.1 


The system helped me 
to practise useful skills 


□ 


□ 


□ 


5.2 


The system helped me 
to find useful information 


□ 


□ 


□ 


5.3 


1 enjoyed using the system 


□ 


□ 


□ 


5.4 


The information in the 
system was clear 


□ 


□ ■ 


□ 


Are 


you being assessed on how to use the system? 






6.1 


Yes 






□ 


6.2 


No 






□ 


6.3 


Not sure 






□ 



7. Is what you produce from the system (e.g. a print out) 
going to be assessed? 



7.1 Yes n 

7.2 No □ 

7.3 Not sure D 

8. Is what you are getting from the system going to be useful 
to you in an assessment? 

8.1 Yes □ 

8.2 No □ 

8.3 Not sure D 

9. Do you understand why you are using the system? 

9.1 Yes n 

9.2 No □ 

9.3 Partially □ 



10. If your answer was 'yes\ please say briefly why you are using it: 



11. What did you think was the best aspect of using the system? 
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12. What was the worst aspect of using the system? 



13. Overall, do you think the system helped you to learn? 

13.1 Yes □ 

13.2 No □ 

14. Please give reasons for your answer: 



Your general use of computers 

This section is to help us to find out about your overall 
level of experience, skill and familiarity with computers. 

15. Do you use computers when you are: 

please tick all that apply 



15.1 


At college/the learning centre 


□ 


15.2 


At home 


□ 


15.3 


At work 


□ 


1 5.4 In other centres or internet cafes 

How often do you use computers generally? 


□ 


16.1 


Daily 


□ 


16.2 


Weekly 


□ 


16.3 


Monthly 


□ 


16.4 


Rarely 


□ 


16.5 


Never 


□ 



17. How would you describe your own use and understanding of ICT? 

17.1 Expert □ 

17.2 Regular and confident user □ 

17.3 Intermediate and learning quickly and enthusiastically □ 

17.4 Beginner or occasional user Q 

17.5 Little or no experience of ICT Q 
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18. Piease expiain, if you can, why you have this ievei of expertise: 



18.1 


Self taught 


□ 


18.2 


On-the-job training 


□ 


18.3 


Training course 


□ 


18.4 


IT career background 


□ 


18.5 


Other 


□ 


Do you enjoy using computers? 




19.1 


Yes 


□ 


19.2 


No 


□ 


19.3 


Not sure 


□ 



20. Piease give reasons for your answer to question 19: 



21. Which of the foiiowing applications/software had you used before 
starting your iearning programme, and which have you used during 
your programme? (piease tick - NB you should tick both coiumns 
if you used the software both before and during your programme) 



Application/software 


Used before 


Used during 
programme 


Word processing 






Spreadsheets 






CD-ROMs 






Internet 






E-mail 






Databases 







22. Many thanks for taking the time to fiii in this questionnaire. If you 
have any other comments, please write them in the space below: 
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Staff questionnaire 
template 



Introduction 

This questionnaire is intended for use as part of an evaluation process 
involving collection and comparison of feedback from the major stakeholders 
(typically learners, teaching staff and managerial staff) in an ICT project 
or strategy. To ensure maximum benefit from the evaluation process, 
please complete this questionnaire fully and frankly. 



Context 

1. Name: 

2. Job title: 

3. Role in this project: 

4. Project title: 

5. Brief description of the project: 



6. Start date: 

7. Status of the project 

please tick one only 



7.1 Flagship □ 

7.2 Pilot □ 

7.3 Response to invitation to tender □ 

7.4 Partof whole-organisation ICT strategy implementation □ 

7.5 Local (department/faculty/school) initiative O 

7.6 Ad hoc □ 
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8 How would you describe senior managers’ commitment 
to the project? 



8.1 Championed by senior management O 

8.2 Actively interested, supportive and □ 

have ensured adequate resources 

8.3 Some interest and support when approached □ 

8.4 Have approved project but will take little interest □ 

until success demonstrated 

8.5 Unaware of project or not interested □ 



9. How would you describe the delivery team’s commitment 
to the project? (e.g. lecturers, tutors, librarians, 
learning assistants involved) 



9.1 Feel ownership for the project and Q 

fully committed to its success 

9.2 Believe the project Is Important and will do □ 

their best to ensure success 

9.3 Interested and will try to dedicate sufficient time □ 

9.4 Have been asked to participate but do not consider it a priority □ 

9.5 Only participating because required to CH 

10. Have there been any changes in the organisation since the start of 
the project that may have affected its outcomes or timescales? i.e. 

10.1 Organisation-wide reorganisation/restructuring □ 

10.2 Changes to senior management team □ 

10.3 New ICT strategy □ 

10.4 Change in management responsible for the project □ 

10.5 Staff turnover in the delivery team □ 

10.6 Other, please specify: 



11. Have you used ICT materials to assist Yes O No O 

your own learning? 

If yes, please give details and Indicate whether or not 
you found this method of supported learning effective. 



12. How would you describe your own use and understanding of ICT? 



12.1 Expert □ 

12.2 Regular and confident user O 

12.3 Intermediate and learning quickly and enthusiastically □ 

12.4 Beginner or occasional user □ 

1 2.5 Little or no experience of ICT □ 
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13. How has the organisation helped you to develop 
your use and understanding of ICT? 



13.1 


Individual tailor-made CPD programmes 


□ 


13.2 


External courses 


□ 


13.3 


In-house courses 


□ 


13.4 


Self-paced flexible learning materials 


□ 


13.5 


On-line learning 


□ 


13.6 


Other, please specify: 





How 

14.1 


often did the delivery team meet? 

Daily 


□ 


14.2 


More than once a week 


□ 


14.3 


Weekly 


□ 


14.4 


More than once a month 


□ 


14.5 


Monthly 


□ 


14.6 


Quarterly 


□ 


14.7 


Occasionally 


□ 


How did the delivery team feed back to the management? 

please tick all that apply 

15.1 Face-to-face meetings 


□ 


15.2 


E-mail 


□ 


15.3 


Web/intranet 


□ 


15.4 


Paper reports 


□ 



16. In the table below, list the programme(s) where you used ICT 
materials (title, level, number of learners, etc.) and explain how 
you prepared the learners for using the materials (preparation 
may include checking learners’ IT awareness, going through 
the project brief, making outcomes explicit, explaining how 
the learners can get help, etc.): 



Title of 
programme 


Level 


Number 
of learners 


Preparation 
of learners 



































Evaluating ICT projects and strategies 



O aluating ICT projects and strategies 

ERIC 





feda 




TheOpen 

University 



quali^ 

learning 

skills 



Resources, access and support 

17. How did learners access the computing facilities? 





17.1 


In classrooms 


□ 




17.2 


In college-based learning centres 


□ 




17.3 


In community-based learning centres 


□ 




17.4 


From work 


□ 




17.5 


From home 


□ 


18. 


Was 


the hardware reliable? 






18.1 


Yes 


□ 




18.2 


Mostly 


□ 




18.3 


To some extent 


□ 




18.4 


No 


□ 


19. 


Was 


the software reliable? 






19.1 


Yes 


□ 




19.2 


Mostly 


□ 




19.3 


To some extent 


□ 




19.4 


No 


□ 


20. 


Please describe any problems you or your learners had 
with the computing facilities: 





21. Was the technical support adequate? 




21.1 


Yes 


□ 


21.2 


Mostly 


□ 


21.3 


To some extent 


□ 


21.4 


No 


□ 



22. Please describe any problems you or your learners had with the 
level or quality of technical support: 
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23. Did you help to select the software/application for your project? 

Yes □ No □ 



If yes, how did you make your selection? 
please tick all that apply 
Did you: 

23.1 Use a formal evaluation method/tool developed in-house □ 

23.2 Use a formal evaluation method/tool developed externally Q 

23.3 Apply standard quality criteria □ 

23.4 Consider acceptability with regard to equal opportunities CH 

23.5 Seek to cater for learners with learning difficulties or disabilities D 

23.6 Seek to cater for a variety of preferred learning styles CH 

23.7 Other, please give details: 



24. Are you satisfied with the software/applications? 



24.1 


Yes 


□ 


24.2 


Mostly 


□ 


24.3 


To some extent 


□ 


24.4 


No 


□ 


25. Please explain your answer: 





26. 


Were 


there enough resources to support your project? 






26.1 


Yes 


□ 




26.2 


Mostly 


□ 




26.3 


To some extent 


□ 




26.4 


No 


□ 


27. 


Please outline some shortfalls: 





28. How did you support your learners while they were 
using the ICT materials? 

28.1 Used them In classroom 


□ 


28.2 


Tutor available if needed 


□ 


28.3 


Progress meetings 


□ 


28.4 


E-mail 


□ 


28.5 


On-line conferencing (e.g. FirstClass) 


□ 


28.6 


Phone 


□ 


28.7 


Other, please specify: 
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29. How did you monitor your learners’ progress? 

29.1 Pre and post test Q 

29.2 Progress monitoring built into the system used CD 

29.3 Log books (paper or electronic) Q 

29.4 On-line tutorials D 

29.5 Face-to-face meetings Q 

30. How did you feed back to your learners on their progress? 

30.1 Feedback provided by system direct to learner Q 

30.2 E-mail □ 

30.3 Paper reports Q 

30.4 Progress meetings □ 

31. How did you give your learners the opportunity 
to feed back on the project? 

31.1 Questionnaire CD 

31.2 Focus group Q 

31.3 Interview D 

31.4 E-mail D 

31.5 Web-based discussion group D 

31.6 Feedback buttons □ 

31.7 Other, please specify: 



32. How much did your learners appear to enjoy the project? 

Estimate the percentage of learners in the following categories: 



32.1 Really enjoyed [ 

32.2 Enjoyed [ 

32.3 Non-committal [ 

32.4 Didn’t enjoy [ 



% 



] 

] 

] 

] 



33. How do you feel this group of learners compares with your past 
learners in terms of how much and how quickly they learned? 



33.1 Much better □ 

33.2 Better □ 

33.3 About the same D 

33.4 Not as good □ 

33.5 Far worse D 

33.6 Can’t tell □ 

33.7 Not applicable Q 



34. Please give reasons for your answer: 
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35. What do you think was the best aspect of the project for the learners? 



36. What do you think was the worst aspect of the project for the learners? 



37. Have you had the opportunity to discuss the strategic aims 
of the project with your manager? 



37.1 Yes □ 

37.2 No □ 

38. Have you got a copy of the strategic aims of the project? 

38.1 Yes □ 

38.2 No □ 



39. If yes, how well do you think your part in the project 
contributed to the strategic aims? 

(ratings based on FEFC inspection grades) 



1 = outstanding achievement □ 

2 = good achievement □ 

3 = satisfactory achievement □ 

4 = less than satisfactory achievement □ 

5 = poor achievement □ 



40. Give reasons for your answer: 
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41 . In the table below, list the specific aims of your project. Rate how 

well the project achieved each aim using the 5 scale in question 39. 
Give reasons for your ratings (include concrete evidence as far as 
possible) and indicate how you plan to improve any unsatisfactory 
ratings. The first two rows have been filled in as an example: 



Specific aims 


Rating 


Reasons 


Planned improvements 


To contribute to 
the improvement 
of basic skills 
of learners 


2 


Improved achievement 
on assessed work 


Further improvement 
of learner materials 
to support basic 
skills development 


To contribute 
to improved 
teamwork skills 
among the learners 


4 


Poor access 
(reported on the 
learner questionnaires) 
to the e-mail system 
discouraged learners 


Technical assistance 



































42. Describe any specific problems you had evaluating the project: 



43. Any other comments? 



Thank you for completing this questionnaire. 
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Manager questionnaire 
tempiate 



Introduction 

This questionnaire is intended for use as part of an evaluation process 
involving collection and comparison of feedback from the major stakeholders 
(typically learners, teaching staff and managerial staff) in an ICT project or 
strategy. To ensure maximum benefit from the evaluation process, please 
complete this questionnaire fully and frankly. 



Context 

1. Name: 

2. Job title: 

3. Role in this project: 

4. Project title: 

5. Brief description of the project: 

6. Start date: 

7. Status of the project 

please tick one only 



7.1 


Flagship 


□ 


7.2 


Pilot 


□ 


7.3 


Response to invitation to tender 


□ 


7.4 


Part of whole-organisation ICT strategy implementation 


□ 


7.5 


Local (department/faculty/school) initiative 


□ 


7.6 


Ad hoc 


□ 
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8. How would you describe senior managers’ commitment 



to the project? 

8.1 Championed by senior management □ 

8.2 Actively interested, supportive and have □ 

ensured adequate resources 

8.3 Some interest and support when approached □ 

8.4 Have approved project but will take little interest □ 

until success demonstrated 

8.5 Unaware of project or not interested □ 



9. How would you describe the delivery team’s commitment 
to the project? (e.g. lecturers, tutors, librarians, 
learning assistants involved) 



9.1 Feel ownership for the project and □ 

fully committed to its success 

9.2 Believe the project is important and □ 

will do their best to ensure success 

9.3 Interested and will try to dedicate sufficient time □ 

9.4 Have been asked to participate but do not consider it a priority □ 

9.5 Only participating because required to □ 

10. Have there been any changes in the organisation since the start of 
the project that may have affected its outcomes or timescales? i.e. 

10.1 Organisation-wide reorganisation/restructuring □ 

10.2 Changes to senior management team □ 

10.3 New ICT strategy □ 

10.4 Change in management responsible for the project □ 

10.5 Staff turnover in the delivery team Q 

10.6 Other, please specify: 



11. Have you used ICT materials to assist your own learning? 

Yes □ No □ 

if yes, please give details and indicate whether or not 
you found this method of supported learning effective. 



12. How would you describe your own use and understanding of ICT? 



12.1 Expert □ 

12.2 Regular and confident, user □ 

12.3 Intermediate and learning quickly and enthusiastically □ 

12.4 Beginner or occasional user □ 

12.5 Little or no experience of ICT □ 
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13. How would you describe use the delivery team’s 




and understanding of ICT? 

13.1 Experts 


□ 


13.2 Regular and confident users 


□ 


13.3 Intermediate and learning quickly and enthusiastically 


□ 


13.4 Beginners or occasional users 


□ 


13.5 Mixed-ability team 


□ 


14. How does the organisation meet the staff development needs 




of the delivery team? 

14.1 Individual tailor-made CPD programmes 


□ 


14.2 External courses 


□ 


14.3 In-house courses 


□ 


14.4 Self-paced flexible learning materials 


□ 


14.5 On-line learning 

14.6 Other, please specify: 


□ 


15. How often did the delivery team meet? 




15.1 Daily 


□ 


15.2 More than once a week 


□ 


15.3 Weekly 


□ 


15.4 More than once a month 


□ 


15.5 Monthly 


□ 


15.6 Quarterly 


□ 


15.7 Occasionally 


□ 


16. How did the delivery team feedback to the management? 




please tick all that apply 




16.1 Face-to-face meetings 


□ 


16.2 E-mail 


□ 


16.3 Web/Intranet 


□ 


16.4 Paper reports 


□ 


Resources, access and support 

17. How did learners access the computing facilities? 




17.1 In classrooms 


□ 


17.2 In college-based learning centres 


□ 


17.3 In community-based learning centres 


□ 


17.4 From work 


□ 


17.5 From home 


□ 
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18. Did learners have any problems accessing 

the computers? Yes D No D 

If yes, please give details 



19. Did learners have any problems getting 

technical support? Yes O No O 

If yes, please give details 



20. Did staff have any problems accessing 

the computers? Yes Q No Q 

If yes, please give details 



21. Did staff have any problems getting 

technical support? Yes □ No O 

If yes, please give details 



22. Did you help to select the software/application 

for your project? Yes Q No Q 

If yes, how did you make your selection? 
please tick all that apply 
Did you: 

22.1 Use a formal evaluation method/tool developed in-house D 

22.2 Use a formal evaluation method/tool developed externally □ 

22.3 Apply standard quality criteria [Zl 

22.4 Consider acceptability with regard to equal opportunities □ 

22.5 Seek to cater for learners with learning difficulties or disabilities CH 

22.6 Seek to cater for a variety of preferred learning styles Cl 

22.7 Other, please give details: 



23. How was the project funded? 

23.1 Fully externally funded Cl 

23.2 Partially externally funded Q 

23.3 Funded from central internal budget Cl 

23.4 Funded from local (e.g. department/faculty/school) budget Cl 
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24. To what extent did the funding support the implementation? 

24.1 Fully □ 

24.2 Adequately O 

24.3 Less than adequately O 

24.4 Inadequately O 

25. What are the implications of the way the project was funded 
in relation to its long-term future? 

25.1 Long-term future secure □ 

25.2 Possibility of project continuing if seen as successful Q 

25.3 Project unlikely to be able to continue O 

when initial funding ceases 

25.4 Project will cease when funding ceases □ 



Outcomes 

26. In the table opposite, list, in order of priority, the strategic 
aims of the project, rate how adequately they were addressed, 
provide reasons for your ratings and note planned improvements. 

Aims may include those related to: 

• learners e.g. concerning achievement, inclusiveness, 
participation, learner support 

• the organisation and its staff e.g. concerning improving the quality 
of the learning environment, staff development, delivery modes, 
modularisation, cost effectiveness, raising the profile/status 

of the organisation, learner recruitment 

• the wider community e.g. concerning employers’ requirements 
or community needs. 

In considering whether, and to what extent, the aims were addressed it 
may be helpful to refer to the FEFC Self-Assessment Quality Statements 
and to rate how well the project achieved each aim using the 
following scale: 

1 = outstanding achievement 

2 = good achievement 

3 = satisfactory achievement 

4 = less than satisfactory achievement 

5 = poor achievement 

Reasons for ratings should include concrete evidence if possible. 
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The first two rows have been filled in as an example. 



Strategic aims 


Rating 


Reasons 


Planned improvements 


Widen participation 


3 


Small increase in 
working-class males 
attending classes 
supported by ICT 


Improve access to 
supported IT sessions for 
development of IT skills 


Extend the range of 
teaching strategies 
to improve learner 
motivation and 
understanding 


2 


Learner questionnaire 
responses. Retention 
and achievement rate 


Encourage take-up 
by more lecturers 











































27. Describe any specific problems you had evaluating the project 



28. How would you describe the project in terms of value for money? 



28.1 Excellent value □ 

28.2 Good value □ 

28.3 Not very good value □ 

28.4 Poor value □ 



29. What evidence do you have for your answer? 



30. Any other comments? 



Thank you for completing this questionnaire. 



Evaluating ICT projects and strategies 




31 



The questionnaire templates 25 




Customising 
the questionnaire 
tempiates 




The questionnaire templates are provided on CD to facilitate customising them 
for particular ICT projects and a variety of uses of ICT in teaching and learning. 

In the college trials the questionnaire templates generally proved easy to adapt. 
This customising was essential for: 

• enhancing usability (of the learner questionnaire in particular) 

• enhancing the relevance of the evaluation process. 

Electronic scanning 

Data collected via questionnaires are often extracted for computer analysis 
by electronic scanning systems, such as optical character recognition (OCR), 
optical mark recognition (OMR) and intelligent character recognition (ICR). 

If the completed questionnaires are going to be scanned, data entry fields 
(e.g. tick boxes) need to be an appropriate size and format. 

Appropriate language 

The language in the learner questionnaire is designed to be suitable for a 
wide variety of learners but customisation may be helpful for learners with 
weak literacy skills or special educational needs. 



Assessment requirements 

In trials some learners had difficulty answering a question about how use 
of the system contributed to their assessment requirements. This question 
can easily be customised to make more sense to learners. For example: 

• Is the work you are typing going to be marked? 

• Is your tutor going to give you a grade for 
your searching of the World Wide Web? 
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Aims and objectives 

Customisation is essential if the questionnaire is to address the unique aims and 
objectives of a project or strategy. Question 5 in the learner questionnaire template 
asks whether learners agree with some very generalised statements. These should 
be replaced with questions that reflect specific aims and objectives. For example: 

• When evaluating the success of an on-line teacher training programme 
in technology, the customised questionnaire might ask how far the 
learners agree with statements like: 

The NewMedia software package introduced me 
to new ideas for teaching my students. 

After using NewMedia I felt confident in using the Web. 

I now feel confident enough to use spreadsheets in the classroom. 

• When evaluating ICT use on a basic skills course, 
the customised statements for learners might include: 

WordSpell helped me to understand 
the meaning of the following words: 

• manufacture • vehicle • profit. 

I like using WordSpell more than the textbook. 

• For learners on an advanced biology programme, 
customised statements could include: 

PhotoWorld helped me to understand the chemistry of photosynthesis. 

After using PhotoWorld I can describe how changes in 
carbon dioxide levels will affect the rate of plant growth. 

I found information in PhotoWorld that will help me 
write the assessed essay. 

Success criteria and performance indicators 

In some cases senior management, or the project manager, may have developed success 
criteria or a set of performance indicators for the project. In this case customisation of 
the questionnaire templates needs to include appropriate questions relating directly to 
the performance indicators or designed to gauge whether or not the success criteria 
have been met. 

Pro- and post-knowledge testing 

Evaluation managers may feel that it is appropriate in some circumstances to include 
questions to test learners’ knowledge before and/or after their use of ICT learning 
systems or resources to check learning and retention of knowledge. For example, 

. the questionnaire for the learners on the biology programme mentioned above, 
might include: 

• How do changes in carbon dioxide levels affect plant growth? 

(please use at least one example in your answer) 

• Draw a diagram to outline the process of photosynthesis. 
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What is evaluation? 

According to Robson (1993), evaluation can generally be defined as a method 
of assessing the effects or effectiveness of something, typically some innovation 
or intervention, on policy, practice or service. 

What is an evaluation strategy? 

According to Sommerlad (1992), designing an evaluation strategy 
is primarily about making choices. These choices concern: 

• the reasons for evaluating (e.g. to modify and improve materials) 

• the focus of the evaluation (e.g. on learning, motivation, inclusiveness, 
equal opportunities, cost effectiveness, etc.) 

• the method to be used (e.g. experimental) 

• the information required to make the evaluation 

• the specific tools to be used (e.g. questionnaires, interviews, 
observations, pre tests, post tests, etc.) 

• who will do the evaluating 

• how the process will be managed, reviewed and modified 

• how to ensure that the results of the evaluation are used 
to achieve improvements. 

Evaluation methodologies 

Current research identifies three broad evaluation methodologies: 

• quantitative methods based on controlled experiment methodology 

• qualitative methodologies 

• hybrid methodologies. 

Experimental evaluations aim primarily to provide summative evaluations of the 
performance and properties of information and communications technology. In this 
type of evaluation often one group of learners is given access to specific ICT resources 
and another group is not. The learning outcomes for both groups are then compared. 
Interpreting the results of these experiments has proved problematic for several reasons, 
one being that since ICT is not used in isolation it is almost impossible to ascribe any 
change in learning outcome to it. These problems are widely discussed by Elton and 
Laurillard (1979), MacDonald and Jenkins (1979), Draper (1997), Gunn (1997) and 
others. Even without them, ethical issues arise when some learners are deliberately 
denied access to materials that may improve their learning and performance. 



Qualitative evaluations aim primarily to discover what factors are important when 
considering learning in relation to ICT. Qualitative methods often start without pre- 
defined aims and seek to identify and explain problems, issues and features during use 
of ICT materials (Oliver and Conole, 1998). They are particularly useful for formative 
evaluation of software used in real teaching situations. Methods include observations, 
interviews with participants, questionnaires and document analysis. 

Hybrid models (Jones etai, 1996; Gunn, 1997; Draper et al.^ 1996) fall 
between these two extremes and aim to help teachers and organisations make 
better use of ICT through cycles of evaluation and modification. The rest of this 
section outlines four hybrid models that have informed the development of the 
evaluation tools provided in this manual. 

Open University framework 

The Open University framework was developed over 15 years of evaluation work 
at the Open University. It uses qualitative and quantitative techniques, information 
from a variety of sources and has three main dimensions: 

• context refers to the aims and rationale behind the use of the technology 

• interactions explores the learners’ interactions with the technology 

• outcomes examines the learner learning that results from interaction 
with the technology. 

The following table summarises the rationale behind each dimension, 
the data collected as part of this and the methods used to collect the data. 



The Open University framework for evaiuation 





Context 


Interactions 


Outcomes 


Rationale 


• Both the past 
evaluations carried 
out by the OU and 
the literature suggest 
that the context of 
the ICT must be 
considered 


• We need to look 
at interaction 
to focus on the 
learning process 


• Learning outcomes 
must be considered to 
try to assess the effec- 
tiveness, but we have 
also argued for the 
importance of effec- 
tive outcomes such as 
changes in learners’ 
perceptions and 
attitudes 


Data 


• Designers’ and 
course teams’ aims 


• Records of 
learner interactions 

• Learner diaries 
and on-line logs 


• Measures of learning 

• Changes in learner 
attitudes and perceptions 


Methods 


• Interview ICT 
designers and 
course team 
members 

• Analyse 
policy documents 


• Observation 

• Diaries 

• Video/audio 
computer recording 


• Interviews, 
questionnaires 
and tests 



Source: Jones etal.^ 1996 
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Teaching with independent learning technologies 

The teaching with independent learning technologies (TILT) model was developed 
by Draper etaL (1996). Their remit was to evaluate a diverse range of computer-aided 
learning (CAL) in use in a university setting. This method can be described at two levels: 
outer and inner. 

The outer method starts with one or more meetings between evaluators, 
teaching staff and developers to: 

• establish teaching staff goals for evaluation 

• elicit the learning aims and objectives to be studied 

• elicit the classroom provision, course integration, and other features of the 
teaching situation, and the support surrounding the courseware itself, e.g. how 
the material is assessed, whether it is a scheduled teaching event or open access 

• establish options for classroom observation, questionnaire administration, 
interviews, etc. 

The evaluators, teaching staff and developers then examine the material to identify 
where assessments, learning quizzes or other measures of learning gains could be 
incorporated. The teaching staff and developers incorporate these, and the evaluators 
and teaching staff finalise a design for the study. The classroom study then occurs, and 
the evaluators draw up a preliminary report which is distributed to the teaching staff. 
Their comments contribute substantially to the final report. 

The inner method concerns the instruments used. A large study may use all the 
following: computer experience questionnaires, task experience questionnaires, 
evaluator observations possibly supported by videotape, learner confidence logs, 
knowledge quizzes, post-task questionnaires, focus groups or interviews, 
learning resource questionnaires and post-course questionnaires. 
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The BP evaluation of learning technologies model 

The BP evaluation of learning technologies model was developed by Martin Oliver 
and his team at the University of North London as part of the BP evaluation of learning 
technologies project. The framework involves several processes, not all of which will be 
appropriate for any given study. The following table outlines the stages of evaluation 
plus the people involved and the evaluation methods at each stage. 



The BP evaluation of learning technologies model 



Intention 


Involves 


Method 


Select appropriate resources, or 
identify appropriate uses for resources 


Practitioners 


Interview 


Specify assessment criteria 
and learning outcomes 


Practitioners 


Interview 


Identify factors that may 
influence learning outcomes 


Researchers 
and practitioners 


Interview and 
literature review 


Gather background information 
about course and content 


Institution 


Requests for 
information 


Gather background data on subjects 


Learners 
and institution 


Survey and requests 
for information 


Gather data on 
initial level of knowledge 


Learners 


Pre test 


Assess effects of 
interventions such as: 

• Condition 1 (e.g. lecture based) 

• Condition 2 (e.g. distance learning) 

• Condition 3 (e.g. open learning) 
and so on 


Learners and 
practitioners 
examples of work 


Observation, 
activity logs, 
examples of work 


Gather data on 
final level of knowledge 


Learners 


Post test 


Assess feedback on material 
and learning experience 


Learners 


Survey and 
interviews 


Gather data on 
long-term level of knowledge 


Learners 


Delayed post test 


Assess time commitments 
and involvement of tutors 


Staff 


Observations 
and interviews 



Source: Oliver, 1997 
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International evaluation projects and methods 

USA 

The Flashlight programme, part of the TLT Group in the USA, has been developing 
evaluation tools since 1994 to help educators answer the most common questions 
about educational technology. First developed was the Current student inventory ^ 
a collection of questionnaires that can be pulled apart and re-amalgamated to suit 
every evaluation, as far as students are concerned. This inventory looks, in part, 
at the effectiveness of educational technology. In 1999, Flashlight released the 
Cos^s analysis handbook^ which helps users build an economic model to measure 
the costs of educational technology. The costs and the effectiveness of educational 
technology have been approached from two directions but are still regarded as two 
separate entities. Another disadvantage of the Flashlight tool is that the programme 
encourages individual use of all the tools and so the data cannot be compared across 
the sector. The Flashlight programme has been used extensively in American junior 
colleges and is now entering the UK evaluation arena. The Joint Information Systems 
Committee (JISC) has recently announced funding for a UK Flashlight consortium 
to test the methodology in UK higher education and to assess its suitability 
as an evaluation tool. 

Australia 

In a framework devised for the evaluation of educational and training programmes 
in commerce and industry, James Athanasou (1998) proposed a six-stage ‘holistic' 
evaluation process: 

• Is the programme or service ethical? 

• To what extent do the programmes or services cover those who are most in need? 

• What are the costs, benefits and utilities of the programme or service? 

• Did the programme or service achieve its key objective/s? 

• What is the net effect of the programme or service? 

• To what extent have the perspectives and interests 
of all stakeholders been considered and met? 

Athanasou also identifies five barriers to evaluation: 

• lack of standards 

• difficulty in measuring the evaluation of training 

• insufficient staff 

• difficulty isolating behaviours that changed as a result of training 

• lack of expertise in evaluation methods. 
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Key evaluation design issues 

Some general points regarding evaluation design emerged from the studies: 

• Many factors affect the successful use of ICT. 

• Any educational evaluation is strengthened if data are gathered 
from as many different sources as possible and conclusions 
are drawn by a process of triangulation. 

• Evaluation should be based around each 
staff member’s/manager’s stated objectives. 

• Evaluation should involve real learners in real situations, 
that is, the evaluation should take place in context. 

• Open-ended questions should be included to investigate unpredicted issues. 

• Strategies should be developed to promote ownership of the evaluation process 
by those directly involved. 

Counting the costs 

Charlotte Ash and Professor Paul Bacsich from Sheffield Hallam University (SHU) 
have promoted an holistic approach to evaluation and believe that: 

• evaluation should start at the beginning of any initiative 

• measurement of effectiveness should go hand in hand 
with measurement of costs. 

The SHU Costs of Networked Learning (CNL) costing methodology takes into account 
every step of the development process, from planning buildings and IT infrastructure, 
to the maintenance of learning materials. The assertion is that evaluation of effectiveness 
must be built upon the same framework. Ash and Bacsich believe that, to be meaningful, 
evaluations must take account of both cost and effectiveness for all stakeholders. 






.-^aluating ICT projects and strategies 



3S 



Analysis guidance 
template 




The following is a checklist of key questions for consideration when: 

• analysing responses to the questionnaires developed 
from the templates in this manual 

• developing reports 

• developing action plans. 

Please note Before using this template it should be customised to include 
the new or amended questions arising out of customisation of the learner, 
staff and questionnaire templates. 



The learner questionnaire 

General information about the respondents 

• How many respondents? What programme/s? Age ranges? 

• What significance might the profile have? (e.g. possible inclusion of 

higher or lower than average numbers of early adopters of new technologies) 

• What was the sampling strategy? (e.g. 50 questionnaires were distributed 
to three groups of GNVQ Intermediate learners, 12 questionnaires were 
distributed to one group of adult returners, etc.) 

• How representative was the sample? 

Use and understanding 

• How many were mistaken about any aspects of assessment? 

• How many thought they understood why they were using the software? 

• How many thought they did not? 

• Were any clearly mistaken about the purpose? 

• What emerged as the best aspects? 

• What emerged as the worst aspects? 

• How many thought the activity helped them learn? How many did not? 

• Did any notable comments arise as reasons for the answers? 

• Does the questionnaire help to explain why the learners 
felt the system helped them to learn, or did not? i.e. 

O is there an apparent relationship between this and their general computer use 
O or their awareness of the reasons for the system use? 

O or their perception of the system accessibility and usability? 

O or the statements in Question 5 that relate to 
specific aspects of their use of the system 

• Do the responses suggest that any particular aspects 
of use of the system need to be addressed? 
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The staff and manager questionnaires 

Staff, managers and programmes 

• How many staff and manager respondents are there, 
and what are their roles in the project? 

• How typical are they? 

• Are there any obvious differences between the managers’ 
and teaching staffs’ overall perceptions of: 

o the project 

O management commitment 
O team commitment? 

If so, are any of these significant? 

• Do the project team members see themselves as generally confident 
about ICT use? and have they used ICT in their own learning? 

• Can this be compared with results of any training needs analysis? 

• If so are results consistent? 

• How will this information feed into arrangements for staff training? 

• Does the range of programmes listed in the table on the staff questionnaire 
reflect the range of programmes targeted by the project or strategy? 

Resourcing and materials 

• Are there issues that need addressing? 

• Are possible solutions indicated? 

• Were equal opportunities and preferred learning styles taken 
into account when selecting and developing the ICT materials? 

Outcomes 

• How positive are the teaching staff about the outcomes of the project? 

• Are their perceptions in line with those of the managers? 

Are they supported by the analysis of the learner questionnaire? 

• What are the perceived strengths and weaknesses of the project? 

• Are the teaching staff aware of the strategic aims of the project? 

• If so, how do they rate their contributions to these aims? 

Consideration of data from additional sources 

• Have you collected information using log books or diaries (paper or on-line)? 

• If so, does this information support or contradict information collected via 
the questionnaires? What is the significance of any apparent contradiction? 

• Have you collected information by observation (via observers or cameras)? 

• If so, does this information support or contradict information collected via 
the questionnaires? What is the significance of any apparent contradiction? 

• Are data available from learner tracking or management information systems (MIS)? 

• If so, do they support or contradict information collected via the questionnaires? 
What is the significance of any apparent contradiction? 

• Does any software used produce statistics, 

learner progress feedback or management information? 

• If so, does this information support or contradict views 
expressed by learners or staff responding to the questionnaires? 

What is the significance of any apparent contradiction? 
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Overall evaluation of project on the basis of all data collected 

• How would you rate the success of the project or strategy? 

• According to the learner, staff and manager responses, and their analysis, 
what improvements could be made to the project or strategy? 

• Which of these improvements are priorities? 

• Do you have an action plan for the prioritised improvements? 

Is this required by a self-assessment or quality improvement programme? 

• How will you involve stakeholders in developing the action plan? 

• How and when will you implement the action plan? 

• How and when will you evaluate any modified project or strategy? 
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Colleges and projects involved 
in trials of questionnaire templates 

The case-study colleges and their ICT projects 

Six colleges were involved in piloting the evaluation questionnaire templates in 1998/99. 
Colleges were selected because they: 

• were enthusiastic about participating 

• were generally working well with ICT (but it was recognised that if any of 
the colleges were having problems with some aspects of implementing ICT 
this would provide a valuable extra dimension to the trial) 

• were different types and sizes of colleges in diverse locations. 

A variety of types and sizes of ICT project with different aims, objectives, timescales and 
levels of staffing were evaluated. The numbers of learners involved ranged from 15 to 700. 
The learners themselves ranged from those with literacy problems to graduates. 

Halton College 

Halton College is based in Cheshire, with its main site in Widnes. It targets the local 
community and the delivery of work-based training. The college has made considerable 
investments in computing and multimedia facilities. It has a large learning centre and a 
multimedia centre with up-to-date technology including high specification Apple Macintosh 
computers and a range of multimedia hardware and software; scanners, digital cameras, 
camcorders, graphics software, authoring tools and video and sound editing facilities. 
The college has a broadcast-quality recording studio, virtual reality equipment and 
video-conferencing facilities. 

The member of staff involved in trialing the evaluation tools had a cross-college role 
and proposed two projects for the evaluation pilot. One involved the use of laptops to 
develop ICT skills in teachers. The other, which was selected for trialing the evaluation 
tools, involved the use of a commercially available authoring package. Course m/b, 
to put materials on-line, to support learners’ learning and develop teachers’ ICT skills. 
Course info is very simple to use and gives teachers the opportunity to produce learning 
materials that include text, video clips, exercises and quizzes. Staff are able to monitor 
use by the learners in their class and to look at both individual and class test results. 
Groups using Course info are connected via the Internet and can therefore form 
discussion groups. 



Luton Sixth Form College 

Luton Sixth Form College was the first sixth form college in the country, and is one of 
three colleges in south Bedfordshire. Most of its provision is aimed at 16 to 19 year-olds. 
The college has 19 subject-based departments grouped into three teaching faculties: 
arts and communication; humanities and business studies; mathematics, science and 
technology. A fourth faculty is responsible for learner services. The college has a strong 
commitment to resource-based learning and its centralised computing facilities include 
a large learning resource centre and three subject-specific resource bases staffed 
generally by library staff and one teacher from the appropriate subject area. 

The college has several large, cross-college ICT projects, two of which were 
earmarked for trialing the evaluation tools. The member of staff was responsible for 
the technical aspects of these projects. One of the projects involved careers software 
for first-year learners on two-year A-level or Advanced GNVQ courses. 

The aim of the project was that the software would be used by approximately 
700 learners per year, in their tutor group sessions, as part of an eight-week series of 
careers sessions. The project was set up in the spring of 1996 and was designed to replace 
a one-hour per week input by the careers staff. The second project, which was described 
as possibly more of a ‘policy decision’, involved switching on access to the Internet on 
100 staff machines and 300 learner machines. This had happened in September 1998, 
and the member of staff responsible was keen to evaluate the impact of this policy decision 
on learners’ work practices. Both projects were used in trialing the evaluation tools. 

Newark and Sherwood College 

Newark and Sherwood College is a relatively small FE establishment in a mainly 
rural area. The immediate area has a population of about 108,000 but the college 
attracts learners from further afield, including overseas. It has four sites in Newark 
and its education and training provision is organised into nine schools of study. 

Since the appointment of the new principal in 1995, the college has invested 
significantly in IT infrastructure, including extensive refurbishment of its 
new learning centre which is open 50 weeks a year. 

The project the college selected to pilot the evaluation tools was the A-level 
resource-based learning project. The project involved about 10 full-time A-level 
courses, 20 members of staff (10 full time and 10 part time) and 100 learners. 

All members of staff were required to cover one six-week block of teaching 
using resource-based learning materials, with an emphasis on ICT. The plan 
was that learners would work independently through the materials and access 
their teachers when they needed help. Some classroom time was still timetabled. 
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Newcastle College 

Newcastle College is a rapidly expanding college which has grown by 50% 
since incorporation. It has a main campus in the west of the city and four other 
sites throughout the city. The college has four faculties: business, management and IT; 
engineering and the built environment; humanities, hospitality and science; visual and 
performing arts. The programmes offered a range from foundation level to first-year, 
non-honours degree courses with vocational courses accounting for nearly 
three-quarters of the college’s provision. 

The staff who were involved in trialing the evaluation tools worked in the 
Business, Management and IT department, where several interesting ICT courses 
were being developed. These included a multimedia taster in key skills, which was being 
developed as part of the University for Industry in collaboration with Gateshead College 
and City of Sunderland College. A telematics course on use of the Internet, e-mail and 
video-conferencing was already running at basic and intermediate levels and the advanced 
level was being set up. The ICT programme on which the evaluation materials was trialed 
was the JEB Certificate in Educational Use of IT. This programme was aimed at school- 
teachers, and was designed to help them use computing technology in the classroom. 

The staff at Newcastle had originally hoped to trial the evaluation materials on the 
key skills project, but because there were delays in getting this on-line they decided 
instead to focus on the JEB Certificate programme. 

Pontypridd College 

Pontypridd College is one of the largest FE providers in South Wales. It has four sites 
and was formed by the merger in 1995 of Pontypridd and Rhondda Colleges. The college 
has approximately 5,000 learners and employs approximately 500 staff (Pontypridd, 1999). 
It offers a wide range of courses, including leisure, construction, art and design, engineering, 
business management and manufacturing technology. The college has a number 
of subject-based workshops and a general IT support unit which is available to 
learners on a ‘drop-in’ basis. 

The project selected for trailing the evaluation tool was a multimedia key skills 
project funded by the Welsh Office for disaffected 14-16 year-olds. The learners 
attended Pen-Y-Dre school and went to Pontypridd College one day a week for their 
GNVQ Foundation Course in Manufacturing. The project involved about 14 learners, 
and used key skills materials that had been developed from paper based materials 
by a full-time technician using Authorware Version 5, 
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Waltham Forest College 

Waltham Forest College is a large general FE college, with mostly vocational courses. 

It has two sites, one in Waltham Forest and one in Chingford. The college has three 
directorates: development, curriculum and administration, and management services. 
There are nine teaching schools in the curriculum directorate: applied science; arts, 
language and teacher training; automobile engineering; business, computing and manage- 
ment; engineering; general education; health and community care; office technology and 
administration; tourism, hospitality and leisure. The college has recently made substantial 
investments in computing equipment. It has well-resourced IT and learning centres, equipped 
with PCs and extensive tutor support, as well as a large English and maths workshop. 

The staff involved in trialing the evaluation tools worked in a variety of departments. 
They were responsible for managing the English and maths workshop, and for delivering 
teacher training programmes. The college is currently developing some ambitious ICT 
projects, two of which were considered for involvement in the trailing. The first of 
these was a basic skills course, designed to develop the numeracy and literacy skills of, 
in the first instance, local carers. The course was FEFC and Basic Skills Agency funded. 
The second was an Edexcel Supporting the learner professional development programme 
whose target group was local teachers; this was designed to integrate ICT into the delivery 
of staff development programmes. There were, however, delays in getting the basic skills 
course on-line, and so the Waltham Forest College team decided to focus on the profes- 
sional development project during the trialing of the evaluation tools. 
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Glossary 



CAL computer-aided learning 

CNL costs of networked learning 

FE further education 

ICR intelligent character recognition 

ICT information and communications technology 

ILT information and learning technologies 

JISC Joint Information Systems Committee 

MIS management information system 

MLE managed learning environment 

NLN National Learning Network 

OCR optical character recognition 

OMR optical mark recognition 

SHU Sheffield Hallam University 

TILT teaching with independent learning technologies 

TLT Teaching, Learning and Technology Affiliate 
of the American Association for Higher Education 

WWW world wide web 
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As the FE sector continues to invest heavily in information 
and learning technology, it becomes increasingly important 
to measure the benefits of ILT and ICT in terms of learning. 
This practical manual describes relevant evaluation 
methodologies, the evaluation process and how 
to analyse and act on feedback from the evaluation. 

It also provides three customisable questionnaire templates 
designed to collect feedback from learners, staff supporting 
learners and managers. The questionnaire templates 
are available on the accompanying CD to facilitate 
easy amendment and use. 




